Methods and systems for internally debugging code in an on-demand service environment

ABSTRACT

A remote debug session for a server group is provided. A server group including multiple servers that perform workload sharing receives a request to debug code executed at the server group. The code is executed on behalf of a client of a database associated with the server group. At least one of the servers of the group initiates a debugging session and establishes a communication connection with the client. The server group maintains the connection open with the client for the duration of the debugging session. Subsequent requests related to the debug session can be handled in a number of ways by the server group, and all communication to the client about processing the requests is through the connection.

CLAIM OF PRIORITY

This continuation patent application claims the benefit of priority ofU.S. patent application Ser. No. 15/263,237, filed Sep. 12, 2016, nowU.S. Pat. No. 9,977,727, issued May 22, 2018, which is incorporatedherein by reference; which is a continuation of and claims the benefitof priority of U.S. patent application Ser. No. 14/636,997, filed Mar.3, 2015, now U.S. Pat. No. 9,465,720, issued Oct. 11, 2016, which isincorporated herein by reference; which is a continuation of and claimsthe benefit of priority of U.S. patent application Ser. No. 13/085,316,filed Apr. 12, 2011, now U.S. Pat. No. 8,972,953, issued Mar. 3, 2015,which is incorporated herein by reference; and further claims thebenefit of priority of U.S. Provisional Patent Application No.61/324,958, filed Apr. 16, 2010, which is incorporated herein byreference.

FIELD

Embodiments of the invention are related generally to databasemanagement, and embodiments of the invention are more particularlyrelated to deleting large amounts of data from a multitenant database.

COPYRIGHT NOTICE/PERMISSION

Portions of the disclosure of this patent document may contain materialthat is subject to copyright protection. The copyright owner has noobjection to the reproduction by anyone of the patent document or thepatent disclosure as it appears in the Patent and Trademark Officepatent file or records, but otherwise reserves all copyright rightswhatsoever. The copyright notice applies to all data as described below,and in the accompanying drawings hereto, as well as to any softwaredescribed below: Copyright© 2011, salesforce.com, inc., All RightsReserved.

BACKGROUND

In many conventional database systems, users access their data resourcesin one logical database, from a user's own systems. Remote access to adatabase includes accessing the database over a network, and consumesnetwork resources as well as resources of the database system beingaccessed. If a client has exclusive access to the database, accessissues are fairly similar between accessing a local database versusaccessing a remote database.

However, access to a remote database shared by multiple users means thatresource use by one user prevents those resources being used by anotheruser. If the remote shared database additionally executes code forusers, remote debugging can decrease performance of the database. Thus,remote debugging in an environment where a user does not have exclusivecontrol over the user means that one user may use a disproportionateamount of database system resources to perform the debugging. Thus,remote debugging may either be disallowed, or the significantperformance degradation may occur. It will be understood that traffic tothe database system related to debugging would not appear the same tothe system as other traffic. Better system performance is achieved whenall traffic can be handled in a standardized way, rather than havingdifferent rules and resource use for different kinds of traffic.

Trying to implement breakpoints, code step through, and evaluationfurther complicates remote debugging due to the traditional exclusivityof execution resources required to implement such debugging techniques.It will be understood that consistency in execution needs to bemaintained for breakpoints and code step through to function properly.Also, traditional distributed execution would create inconsistency inevaluation of the debugging process.

BRIEF DESCRIPTION OF THE DRAWINGS

The following description includes discussion of figures havingillustrations given by way of example of implementations of embodimentsof the invention. The drawings should be understood by way of example,and not by way of limitation. As used herein, references to one or more“embodiments” are to be understood as describing a particular feature,structure, or characteristic included in at least one implementation ofthe invention. Thus, phrases such as “in one embodiment” or “in analternate embodiment” appearing herein describe various embodiments andimplementations of the invention, and do not necessarily all refer tothe same embodiment. However, they are also not necessarily mutuallyexclusive.

FIG. 1 is a block diagram of an embodiment of an environment thatsupports remote debugging in a database system.

FIG. 2 is a block diagram of an embodiment of a remote debuggingenvironment in a server group.

FIG. 3A and 3B represent flow diagrams of embodiments of execution of aremote debugging process.

FIG. 4 is a block diagram of an embodiment of an environment foron-demand database services.

FIG. 5 is a block diagram of an embodiment of an environment foron-demand database services with a multitenant database having multipledata tables.

FIG. 6 represents a flow diagram of an embodiment of remote debugging ina server group environment.

Descriptions of certain details and implementations follow, including adescription of the figures, which may depict some or all of theembodiments described below, as well as discussing other potentialembodiments or implementations of the inventive concepts presentedherein. An overview of embodiments of the invention is provided below,followed by a more detailed description with reference to the drawings.

DETAILED DESCRIPTION

As described herein, remote debugging is provided from a client to aserver group. A server group includes multiple servers that performworkload sharing of operations for multiple different clients. Theresources of the server group are shared among the multiple clients of adatabase that stores data for the clients.

The server group receives a request to debug code for the requestingclient executed at the server group. One of the servers of the groupinitiates a debugging session and establishes a communication connectionwith the client. The one server maintains the connection open with theclient for the duration of the debugging session. Subsequent requestsrelated to the debug session can be handled in a number of ways by theserver group, and all communication to the client about processingrequests related to the debug session is handled through the connectionby the one server.

In one embodiment, the database is a multitenant database system (MTS).As referred to herein, a multitenant database stores data for multipleclient organizations, which are each identified by a tenant ID. An MTSis a database system with a multitenant database. One or more users canbe associated with each tenant ID. The one or more users of each clientorganization access data identified by the tenant ID associated with therespective client organization. The multitenant database is typically adatabase stored and hosted remote from the client organization.Typically, the multitenant database is hosted by an entity (e.g.,company) separate from the client organization. An MTS as describedherein can be said to provide on-demand database services. An MTS istypically a hosted system that provides as much database storage andservice as required by each tenant.

It will be understood that because multiple tenants share the databasesystem resources (e.g., processing, network bandwidth, process threads,database connections, memory, storage), if one tenant or client uses adisproportionate of resources for debugging traffic performance may benegatively affected for other clients. As described herein, the openconnection and the debug session enable a database system to treat thedebugging traffic like any other traffic in the system. Thus, whetherrequests are received for a specific debugging session, or for otheroperations within the system, the system is able to handle theoperations and distribute work within the server group without havingone user tie up resources.

Additionally, the initiating of the debugging session and theestablishing of the communication connection allow for a consistentpoint of communication to the client. Thus, the server group can allowsharing of resources among client and still provide consistency ofsession experience that is needed for debugging. Thus, from auser-experience perspective, the remote debugging in the remote databaseenvironment allows debugging as if the debugging environment were local.

Furthermore, the consistency and the single communication connectionenable the use of break points, step through, and evaluation via aremote debugging environment. Thus, a server group can treat debuggingtraffic like other traffic for purposes of execution, and not have toallocate resources to allow debugging.

FIG. 1 is a block diagram of an embodiment of an environment thatsupports remote debugging in a database system. Environment 100 allowson-demand database services to client 110 over network 120. Client 110represents any type of user device or client device, such as any type ofcomputing device (e.g., desktop, laptop, handheld, mobile phones orother mobile devices), through which a user client accesses server cloud(or server group) 130. Network 120 includes any form of network throughwhich the client device of client 110 can access remote server cloud130. Network 120 includes transmission hardware and lines, processingequipment, and appropriate software systems as is understood in the art.

As described herein, client 110 allows for a user to perform debuggingof code 146 that executes on server cloud 130. Code 146 may be anapplication on data stored in database 150. Thus, code 146 may be“client code” or a client application that is stored locally to servercloud 130, and executed at server cloud 130, remotely from client 110.The client can thus be considered the debugging engine, but with thedebugging processes and threads executing on server cloud 130. In oneembodiment, client 110 includes IDE (integrated development environment,also referred to as integrated debugging environment or integrateddesign environment) 116.

Client 110 requests the debugging session over communication link 112,over network 120. Link 112 represents any communication connection overwhich client 110 would interact with server cloud 130. The connectionmay include any known login, handshaking, key/credential verification,and/or other process to provide access by client 110 to database 150.Client 110 accesses database 150 via operations by server cloud 130,which receives and processes requests from client 110.

Server cloud 130 represents a pool or a coordinated group of servers,such as application servers. It will be understood multiple applicationservers can execute on the same server hardware. There may be multiplephysical server hardware elements, and multiple application serversimplemented on the hardware. Servers 132-0 through 132-N represent theservers that receive and process requests from client 110, where N isany integer number. Servers 132-0, . . . , 132-N perform workloadsharing, where operations are distributed among the servers forexecution. Distribution of workload can be managed by a scheduler (notexplicitly shown), and/or one of the servers may act as a master thatdistributes work. There may be queues for each server where operationsare distributed, or a common queue that is accessed in a shared fashion(e.g., when a server completes a task it takes the next operation fromthe queue).

In one embodiment, servers 132-0, . . . , 132-N represent a number ofApex processes or instances, each considered a server for purposes ofdebugging. A request for debugging is provided to server cloud 130, andone of the servers processes the request from client 110. The server maybe randomly assigned (e.g., whichever server is randomly assigned therequest as a task). Alternatively, there may be a single serverdesignated for receiving such requests. The processing server registersa debugging session (e.g., for the user or for the client device). Thedebugging session is assigned an identifier (ID). One of the servers ofthe group establishes communication connection 114 with client 110,which will be maintained open for the duration of the debugging sessionto send communication to the client regarding the operations of thedebugging session.

In one embodiment, the ID of the debugging session is associated withthe server that establishes and maintains connection 114. The server maybe the registering server, or another server, depending on theimplementation of server cloud 130. Any of servers 132-0, . . . , 132-Nmay perform operations related to the debugging session, and then directcommunication related to the debugging session back to the serverassociated with the debugging session ID, which is the server thatmaintains connection 114 with client 110. Thus, all debugging requestscan be received and processed just like any other traffic received bythe database system (i.e., database 150 and any servers associated withit).

In one embodiment, a separate debugging stack is created with anassociated debugging thread started when the debugging session wasinitiated. The thread (and its associated stack) could be associatedwith the debugging session ID, and all state information could bepersisted on the stack. Thus, any server that performs an operationrelated to the debugging session could access state related to thedebugging session. Alternatively, the state information could bemaintained in a mem cache shared among the servers. Debug session 142represents the debug or debugging session referred to above. Debugsession 142 is registered and assigned ID 144.

Database 150 includes hardware storage resources, and softwarecomponents that execute on processing resources to enable access to thedata. A database may be considered a structured approach to storing andaccessing data stored in storage resources (such as implementing arelational model, object oriented model, or an object-relational modelfor data storage). There are many different implementations of adatabase, and the specific implementation is not relevant to thediscussion herein. The database system includes the storage andprocessing resources, including interface hardware and software (such asphysical network connections, network protocol stacks, securityelements, and other components for interfacing with the stored data).

Communication connection 114 can be considered a “projected” serversocket in the sense that one server (for example, server 132-0)establishes and maintains connection 114, and any other server 132-1, .. . , 132-N that executes tasks related to debug session 142 sendsinformation back to server 132-0 for communication to client 110.

In one embodiment, in the debugging session context, the serverestablishing connection 114 with client 110 connects to IDE 116 ofclient 110, rather than having the client establish the connection torequest status. The server can then run an agent that exposes a port andallows the IDE to connect to the exposed port. The server then maintainsthe connection open until debug session 142 is closed. While other typesof connections may be used, the use of a socket-based connection allowslayering the connection over HTTP (hypertext transfer protocol).

While specific reference is made above to FIG. 1, in general,embodiments of the invention relate to remote debugging from a servergroup environment that executes the code to be debugged by a remoteclient.

The server group registers a session, which indicates debugging isrequested for a particular user. In one embodiment, one of the serversis designated as a “hold open” server, which holds open a connectionback to the client device, and more particularly, to an IDE of theclient device. In one embodiment, the “hold open” server executes acommunication thread to maintain the communication connection.

With the communication connection maintained, any one of the servers canperform actions related to debugging, and the results of the operationsare sent back to the “hold open” server that controls the communicationconnection. Thus, any request from a user is funneled back to the “holdopen” server for communication back to the client device or the user.The “hold open” server may be considered a focal server. In oneembodiment, it is the server that handles the original request from theclient device. In one embodiment, whenever debugging code is executed(e.g., whenever Apex execution starts), the executing server will lookfor the registered user, and if the user is found, the executing serverwill connect back to the “hold open” server, which may also be theregistering server, which then communicates back to the client.

For any server to be able to perform the debugging operations, debuggingstate should be shared among the server group. In one embodiment, theshared state is stored in mem cache. Other implementations could be usedwhere information does not need to be shared because specific serverassignments are made. For example, there could be a pool of serversdedicated to maintaining IDE connections, and they could implement anIDE connection stack.

In one embodiment, communication between the client and the server grouprelated to debugging is encrypted. For example, SSL could be used tosecure the communication. In one embodiment, the server or serversmaintaining the communication connection can provide a unique ID back tothe client to identify the communication (e.g., similar to a session ID,but only for communication—in fact, it could be used in addition to asession ID).

In one embodiment, the debugging protocol (e.g., DBGp) has a way todefine breakpoints. The breakpoint operations are modified in that abreakpoint can be registered in the same state as the communicationconnection, and be connected back to the server or servers maintainingthe communication connection. In one embodiment, when debug executionpauses, a timeout mechanism starts. If a timeout occurs, it may disableall breakpoints and finish the debug operations to prevent a hungprocess.

In one embodiment, a database system that supports remote debugging asdescribed herein can be created by employing Eclipse IDE, EclipseDynamic Language Toolkit (DLTK), and Apex Events. It will be understoodthat other languages and conventions could also be used. Referring tothe example given, an Eclipse IDE can be implemented with a plugin thatsupport and displays Apex. The DLTK is a tool that provides extensibleframeworks designed to reduce the complexity of building full featureddevelopment environments for dynamic languages (e.g., PHP and Perl), ordevelopment environments for other languages (e.g., Ruby, JavaScript,Python). The DLTK's debugging is based on XDebug's DBGP (debuggingprotocol—usually referred to as DBGp protocol or DBGP protocol). Theprotocol is a standard socket based mechanism for a debugging engine(e.g., Apex) to interact with an IDE (integrated developmentenvironment, such as Eclipse) in a relatively language-neutral fashion.

Apex Events provides event support, provides a mechanism for hooking andrunning Apex code with an external process in a generic manner, and ispart of a generic debugger and logging framework for Apex. It will beunderstood that all trademarks use here and elsewhere are meant only toidentify exemplary elements currently known; all trademarks are theproperty of their respective owners.

In one embodiment, creation of a debugging session includes creating amain thread that remains open for the life of an application server, orone of the servers of the server group. The main thread may beconsidered a “fake” thread in that it may not have any work to perform.However, it can maintain the session in the server and not get closed atthe conclusion of a set of threads. Operation of a debugger ofteninvolves closing a debugging launch after completing all threads. In thecase where different application servers may execute differentoperations, the session may need to be kept open for longer than theexecution of certain operations associated with the debugging session.

In one embodiment, the remote debugging environment as described hereinincludes the execution of two threads, one to execute the debugger, andone to maintain communication with the client device. In an embodimentwhere an Apex debugging thread is used, there may be one Java thread toexecute Apex (an Apex thread), and another Java thread to providecommunication to the client device (an IDE thread). Thus, for example,the client can send a pause command via the communication channel, andthe IDE thread signals the Apex thread to stop on the next statement andpause while the IDE thread performs its operations.

In one embodiment, three mechanisms are used to coordinate between theApex and IDE threads and to enable the IDE thread to perform itsoperations while the Apex thread is paused. A first mechanism indicateswhat happens on startup or initialization of the Apex thread. Namely, onstartup the Apex thread connects to the client device, and awaits acommand to begin stepping into the debugging. A second mechanismincludes a stack manager, which may have a method called by every Apexstatement. The stack manager maintains block variables in scope, orvariables related to a lock between the Apex and IDE threads. The stackmanager may turn on a suspend to block the Apex thread and stay in aloop awaiting the IDE thread to continue. The stack manager may attemptto regain a lock every predetermined time period (e.g., every threeseconds) when a suspend lock is turned on, and if it cannot do so aftera longer predetermined time period (e.g., five minutes), it will assumethe communication thread is not functioning properly. If the suspendlock is not turned on, the stack manager may check to determine if thereis a breakpoint. If there is a breakpoint, it pauses the Apex thread andsends a break to the IDE thread, and then waits as above.

In another embodiment, other protocols or debugging engines may be used,in which case the Apex thread may be more generically referred to as adebugging or debug thread, and the IDE thread may be either an IDEthread or a client communication thread.

In one embodiment, most other commands related to the debug environmentoperate while the Apex thread is suspended. For example, contextvariable commands (stack_get, context_get, property_get) use the stackmanager and a container associated with the stack manager to return theappropriate information, and an evaluation command re-parses anexpression in the context of the current statement's container andreturns the result. In one embodiment of a database system, there may bea single thread to a request, which would mean that before the abovecommands are executed, the state of the debug thread should be copiedover into the IDE thread's context.

FIG. 2 is a block diagram of an embodiment of a remote debuggingenvironment in a server group. Environment 200 could be an example of anenvironment according to environment 100 discussed above. Environment200 includes an on-demand database services environment with multitenantdatabase system 230, and allows remote debugging according to anyembodiment described herein. Client 210 is a client device according toany type of client device described herein. Network 220 is a networkaccording to any embodiment described herein. Client 210 sends remotedebugging requests over network 220 to MTS 230, and a server of MTS 230establishes a debugging communication connection to client 210 overnetwork 220.

MTS 230 is a multitenant database system, with multitenant database 250,and a group of application (app) servers 232. Database 250 providesstores data for multiple client organizations each identified by atenant ID and one or more users are associated with the tenant ID. Theclient device or a user associated with the client device is one of theusers associated with the tenant ID. In MTS 230, the one or more usersof each client or tenant organization access data identified by thetenant ID associated with the respective client organization, butgenerally do not access data associated with other tenant IDs. Database250 is remote from client 210 in that it is hosted by an entity ororganization separate from the client organization associated withclient 210. Application server group 232 provides processing servicesfor access to the database 250.

Application server group 232 has associated communication stack 236,which supports the communication connection back to client 210 thatremains open for the duration of a debug session. In one embodiment,application server group 232 has associated shared mem cache 234, whichis a mem cache as understood by those skilled in the art. Shared memcache 234 provides a mechanism for servers that perform workload sharingto share state of various processes.

Processing threads 240 represent threads that may be executing onapplication servers 232. Such threads include processes for clientapplications 242 and/or other client code 244. When client 210 performsdebugging of client applications 242 or other client code 244, debugrequests are also received and processed by application servers 232,just as the applications are also executed by the application servers.In one embodiment, a stack manager (not explicitly shown) checks forbreakpoints and step-through processes of debugging to handle steppingin and stepping out in a way that is smooth within the MTS.

In one embodiment, application server group 232 generates a newprocessing thread 240 for each subsequent service request associatedwith a debug session. Each processing thread can exist for service uponthe request, and then be closed. However, with use of shared mem cache234, or other state sharing mechanism, the state of the debug sessioncan be maintained for all operations executed for the debug session.

In one embodiment, the remote debugging supports the use of breakpoints.A server of group 232 that is processing a request related to abreakpoint can set a breakpoint that is associated with whateverprocessing thread is related to operations that will be interrupted forpurposes of debugging. Thus, a breakpoint can be set to interrupt onlyexecution of a thread of a server handling execution of the debuggingsession, without interrupting execution of other processing threads 240.So, one processing thread can be interrupted but not tie up resources ofthe system. In one embodiment, the breakpoint is associated with atimeout that can be set when the breakpoint is set. Execution of thethread can be resumed without additional breakpoints if the timeoutoccurs (i.e., the full length of the timeout period expires). This canprevent hung threads in MTS 230.

FIG. 3A and 3B represent flow diagrams of embodiments of execution of aremote debugging process. Flow diagrams as illustrated herein provideexamples of sequences of various process actions. Although shown in aparticular sequence or order, unless otherwise specified, the order ofthe actions can be modified. Thus, the illustrated implementationsshould be understood only as an example, and the process can beperformed in a different order, and some actions may be performed inparallel. Additionally, one or more actions can be omitted in variousembodiments of the invention; thus, not all actions are required inevery implementation. Other process flows are possible.

In a swim-lane flow diagram as illustrated, the flow is furtherunderstood by reference to the component that performs the operation. Inthe embodiment of FIG. 3A, client 302 sends a request for remotedebugging to a server group that includes servers 304 and 306. Otherservers may also exist. In this embodiment, server 304 receives andprocesses an initial debug request from client 302, and also processeslater requests associated with the debug session. For example, server304 could be a server dedicated out of the server group to process debugrequests. The server could be dedicated by predetermination, orconfiguration of the server group, and/or could be designated because itrandomly receives the initial request by client 302.

In the embodiment of FIG. 3B, server 304 processes an initial request,establishes a communication connection, and server 306 performsoperations related to a later request. An additional alternative notshown is that server 304 may process the original request and then someother server of the group could establish the communication connectionto client 302. Other alternatives will also be understood by the skilledreader related to variations of one server processing an initialrequest, and either that server or another server handling laterrequests.

Referring again to FIG. 3A, client 302 requests debug of code executedby the server group, 310. Server 304 processes the request, 312, eitherby designation, or by random assignment. Server 304 creates a debugsession in response to the request, 314. Server 304 registers a debugsession, 316. Server 304 then opens a debug communication channel withclient 302 for all communication related to the debug session, 318.Server 304 maintains the debug communication channel open with theclient for the duration of the register debug session, 320.

The break represents any duration of time that may transpire between theinitial request by client 302 to any subsequent request related to thesame debug session. The subsequent request is sent by client 302 as arequest for a debug operation, 330. In one embodiment, server 304processes the request, 332, either by system configuration, or by randomassignment. Server 304 executes the requested debug operation, 334, andthen needs to communicate the results of the operation back to client302. Server 304 initially established the debug communication channelwith client 302, and thus can simply report on the debug communicationchannel, 336.

Referring to FIG. 3B, the same debug request 310, and request processingincluding establishing a debug communication channel are performed asbefore with FIG. 3A. Again, there is a break, which could be any amountof time. In one embodiment, the break is simply the completion of theinitial processing of the request, and additional debug operations arealready pending in the server group.

In the specific embodiment illustrated, subsequent debug operationrequest is sent to the server group, 350, which is processed, 352, byserver 306. In one embodiment, server 306 executes the debug operation,354. In an alternate embodiment, the debug operation is sent to anotherserver (either server 304, or another server not shown) for execution,360. If server 304 performs the requested debug operation, it will againsimply report on the debug communication channel it maintains open withthe client, 358. If any server other than server 304 executes thesubsequently requested debug operation, that server (e.g., server 306 asshown) reports via proxy, 356. Reporting via proxy means thatcommunication is provided to server 304 (the server that holds open thedebug communication channel) to communicate back to client 302.

FIG. 4 is a block diagram of an embodiment of an environment foron-demand database services. Environment 400 includes components relatedto an on-demand database service. Environment 400 includes multitenantdatabase system (MTS) 440 and one or more organizations 410 and 420,which are tenants of the MTS. Each organization can include one or moreusers and/or user devices.

MTS 440 provides on-demand database services for environment 400. Anon-demand database service, such provided by MTS 440, is a databasesystem that is made available to an outside user as needed by the user(e.g., on the demand of the user). Thus, a user does not necessarilyneed to be concerned with building and/or maintaining the databasesystem, and rather can simply store and access data as needed from aremotely operated database system.

In one embodiment, MTS 440 stores information from one or more tenantsinto tables of a common database image or multitenant database (MTDB)450. Accordingly, MTS 440 provides on-demand database service. Adatabase image may include one or more database objects. A multitenantdatabase stores data for various different tenants or organizations in asingle database instance. Resources (such as memory, processing space,processing hardware, and other resources of the database system) areshared or allocated among the different tenants.

Multitenant database 450 includes tenant data 452, . . . , 454. Thetenant data may be divided into different storage areas, which can be aphysical and/or a logical arrangement of data. In one embodiment,multitenant database 450 is accessed via a relational databasemanagement system (RDBMS) or the equivalent, which executes storage andretrieval of information against the database object(s). In oneembodiment, multitenant database 450 is accessed via an object-orienteddatabase management system (OODBMS) or the equivalent. In oneembodiment, multitenant database 450 is accessed via anobject-relational database management system (ORDBMS) or the equivalent.It will be understood that an RDBMS manages data stored in the databasebased on a relational model, where data and data relationships arestored in tables. An OODBMS includes at least some integration of adatabase with an object-oriented programming language, and data isstored in the database in the same mode of representation as is providedin the programming language. An ORDBMS implements both a relationalmodel and an object-oriented model, storing data in tables, and allowingrepresentation of data consistent with a programming language.

Application platform 448 represents a framework that allows applicationsof MTS 440 to execute. Thus, application platform 448 includes thesoftware components (such as an operating system) to allow execution ofthe applications. Hardware platform 460 provides hardware resources toenable the applications to execute on application platform 448, as wellas enabling execution of management or control logic for MTS 440. In oneembodiment, application platform 448 of MTS 440 enables creation,managing, and executing one or more applications developed by theprovider of the on-demand database service, users accessing theon-demand database service via network 430, or third party applicationdevelopers accessing the on-demand database service via network 430.

MTS 440 represents any type of system that may provide on-demanddatabase service. In addition to application platform 448 and hardwareplatform 460, which includes processor resources 462 and memoryresources 464, MTS 440 may include other components. MTS 440 includesnetwork interface 442 to enable user devices to access MTS 440 overnetwork 430. In one embodiment, MTS 440 includes system data 444,program code 446, and process space 449. System data 444 represents dataspecific to the running of MTS 440, rather than being tenant data. It islogically separated from the tenant storage, and may be physicallyseparated (e.g., by designating storage areas or address ranges forsystem data). Program code 446 represents code to implement variousfunctions of MTS 440, which enable the system to provide on-demanddatabase service. Process space 449 represents a framework for executingMTS processes and tenant-specific processes, such as runningapplications as part of an application hosting service. Additionalprocesses that may execute on MTS 440 include database indexingprocesses. It will be understood that MTS 440 may include more or fewercomponents than what is illustrated.

As mentioned above, environment 400 includes organizations 410 and 420,which represent tenants of MTS 440. Each organization may include one ormore individual, and may be an individual or small company, up to alarge corporation or organization. Thus, it will be understood that thenumber of user devices associated with each organization couldpotentially be hundreds or even thousands. Each organization is assigneda tenant identifier (ID) within MTS 440. Each tenant ID could havecertain associated properties for use, depending on how the organizationis configured. User device 422 is associated with organization 420, andaccess MTS 440 under the tenant ID of organization 420. Similarly, userdevices 412 and 416 are associated with organization 410, and access MTS440 under the tenants ID assigned to organization 410.

User devices 412, 416, and 422 may be any machine or system that is usedby a user to access a database user system. For example, any of the userdevices can be a handheld computing device, a mobile phone, a laptopcomputer, a work station, and/or a network of computing devices. Eachuser device can be provided with an on-demand database service from MTS440 via network 430.

Within an organization, users may be further given access privilegesand/or restrictions, as illustrated by data filter 414. As illustrated,user device 416 may access MTS 440 in accordance with whatever access isavailable to organization 410, while user device 412 has additionalrestrictions applied by data filter 414. In one embodiment, data filter414 may additionally or alternatively provide specific user interfacefeatures for user 412 in accessing data from MTS 440.

The users of user devices 412, 416, and 422 may differ in theirrespective capacities, and the capacity of a particular user devicemight be entirely determined by permissions (permission levels) for thecurrent user. For example, where a salesperson is using a particularuser device to interact with MTS 440, that user device has thecapacities assigned to that salesperson. However, an administrator usingthe same user device may have different capacities assigned to thatadministrator. In systems with a hierarchical role model, users at onepermission level may have access to applications, data, and databaseinformation accessible by a lower permission level user, but may nothave access to certain applications, database information, and dataaccessible by a user at a higher permission level. Thus, different userswill have different capabilities with regard to accessing and modifyingapplication and database information, depending on a user's security orpermission level. Such enforcement could occur based on data filter 414,which can filter per device and/or could filter for the entireorganization (e.g., a central filter as opposed to distributedfiltering).

Network 430 represents any network or combination of networks. A networkis generically an interconnection of devices that communicate with eachother. Network 430 can be or include any combination of a LAN (localarea network), WAN (wide area network), telephone network, wirelessnetwork, point-to-point network, star network, token ring network, hubnetwork, or other appropriate configuration. TCP/IP (Transfer ControlProtocol and Internet Protocol) networks are commonly used, such as theglobal internetwork of networks often referred to as the “Internet.”Reference to specific networks in certain examples herein is meant onlyto provide examples, and is not limiting.

In one embodiment, user devices 412, 416, 422 (and other user devicesnot shown) communicate with MTS 440 over network 430 using TCP/IP and,at a higher network level, use other common protocols to communicate,such as HTTP (HyperText Transfer Protocol), FTP (File TransferProtocol), AFS (Andrew File System—a distributed network filesystemusing trusted servers), WAP (Wireless Access Protocol). In an examplewhere HTTP is used, user device 412 might include an HTTP clientcommonly referred to as a “browser” for sending and receiving HTTPmessages to and from an HTTP server at MTS 440 (not specifically shown,but which could be executed on hardware platform 460). Such an HTTPserver might be implemented as the sole network interface between MTS440 and network 430, but other techniques might be used as well orinstead. In one embodiment, the interface between MTS 440 and network430 includes load sharing functionality, such as round-robin HTTPrequest distributors to balance loads and distribute incoming HTTPrequests evenly over a plurality of servers. At least as for the usersthat are accessing that server, each of the plurality of servers hasaccess to data in MTS 440; however, other alternative configurations maybe used instead.

In one embodiment, MTS 440 implements a web-based customer relationshipmanagement (CRM) system. For example, in one embodiment, MTS 440includes application servers configured to implement and execute CRMsoftware applications as well as provide related data, code, forms,webpages and other information to and from user devices (e.g., 412, 416,422) and to store to and retrieve from a database system related data,objects, and webpage content. With a multitenant system, data formultiple tenants may be stored in the same physical database object;however, tenant data is typically arranged so that data of one tenant iskept logically separate from that of other tenants. The logicalseparation prevents one tenant from having access to another tenant'sdata. An express sharing of data among tenants is possible, whichremoves the logical separation. In one embodiment, MTS 440 implementsapplications other than or in addition to a CRM application. Forexample, MTS 440 may provide tenant access to multiple hosted (standardand custom) applications, including a CRM application. User (or thirdparty developer) applications, which may or may not include CRM, may besupported by application platform 448, which manages creation, storageof the applications into one or more database objects and executing ofthe applications in a virtual machine in process space 449 of MTS 440.

In one embodiment, MTS 440 is configured to provide webpages, forms,applications, data and media content to user (client) device to supportthe access by user devices as tenants of MTS 440. In one embodiment, MTS440 provides security mechanisms to keep each tenant's data separateunless the data is shared. More than one MTS may be used. If more thanone MTS is used, the multiple systems may be located in close proximityto one another (e.g., in a server farm located in a single building orcampus), or they may be distributed at locations remote from one another(e.g., one or more servers located in city A and one or more serverslocated in city B).

As used herein, each MTS could include one or more logically and/orphysically connected servers distributed locally or across one or moregeographic locations. Additionally, the term “server” refers to acomputer system, including processing hardware and process space(s), andan associated storage system and database application (e.g., OODBMS,RDBMS, ORDBMS) as is known in the art. It will be understood that“server system” and “server” are often used interchangeably herein.Similarly, a database object described herein can be implemented assingle databases, a distributed database, a collection of distributeddatabases, a database with redundant online or offline backups or otherredundancies, and might include a distributed database or storagenetwork and associated processing intelligence or logic.

FIG. 5 is a block diagram of an embodiment of an environment foron-demand database services with a multitenant database having multipledata tables. Environment 500 includes components related to providing anon-demand database service, and may be one example of environment 400 ofFIG. 4, with additional components shown. Environment 500 includes oneor more multitenant database systems (MTS) 540 and one or more tenantsof the MTS, as illustrated by user device 510. User device 510 isgenerally part of an organization that is the tenant, and user device510 provides a computing device through which access to MTS 540 isavailable. MTS 540 provides on-demand database services for environment500.

Environment 500 may include conventional, well-known elements that areexplained only briefly here. For example, user device 510 (and any otheruser devices through which users access MTS 540) could include a desktoppersonal computer, workstation, laptop, handheld device, cell phone orsmart phone, or any wireless access protocol (WAP) enabled device or anyother computing device capable of interfacing directly or indirectly tothe Internet or other network connection.

User device 510 includes processor 512, which represents one or moreprocessor devices, and may be any combination of one or more processors.Processor 512 provides hardware means to execute programs andapplications on user device 510. Memory 514 represents a memory systemfor user device 510, and may be any combination of one or more memorydevices, short term, and/or long term memory. I/O (input/output) 516represents any type of input and output devices such as keyboards,pointers and controllers, touchscreens, buttons, microphones, or otherinput mechanisms, and monitors, screens, printers, interfaces tonetworks, and/or other output devices.

User device 510 includes network interface 518, which representshardware interconnections and control logic and circuitry to enable userdevice 510 to connect to network 530. Network interface 518 also hasassociated drivers and possibly other software components to allow userprograms to interface with the interconnection hardware. User device 510includes client 520, which represents a program that allows a user ofuser device 510 to access information from network 530, such asaccessing MTS 540. UI 522 represents a user interface component ofclient 520, or a user interface in which information from client 520 ispresented on user device 520. Thus, UI 522 may be integrated with client520, or it may be separate from client 520, but display data related tothe execution of client 520. UI 522 is rendered on display or userinterface hardware or device, which can be understood to be representedby UI 522.

In one embodiment, user device 510 runs an HTTP client as client 520. AnHTTP client may be, for example, a browsing program or a browser, whichmay include a WAP-enabled browser in the case of a cell phone, PDA orother wireless device. The HTTP client allows a user (e.g., subscriberof MTS 540) of user device 510 to access, process, and view information,pages, and applications available from MTS 540 over network 530, basedon permissions and privileges. The user interface device of user device510 can be used to access data and applications hosted by MTS 540, andto perform searches on stored data, and otherwise allow a user tointeract with various GUI (graphical user interface) pages that may bepresented to a user.

Similar to what is discussed above with reference to network 430 ofenvironment 400, network 530 represents any network or group of networksover which access can be provided to MTS 540. Network 530 may includeswitching and/or routing elements, cables, connectors, and othercomponents. In one embodiment, at least part of network 530 is theInternet, referring to a specific global internetwork of networks.However, it should be understood that other networks can be used inaddition to or instead of the Internet, such as an intranet, anextranet, a virtual private network (VPN), a non-TCP/IP based network,any LAN or WAN or other network.

In one embodiment, user devices such as user device 510 (which may beclient systems) communicate with application server 550 to request andupdate system-level and tenant-level data from MTS 540 that may requiresending one or more queries to tenant data storage in database instance580 and/or system data in system database 542. In one embodiment, MTS540 (e.g., application server 550) automatically generates one or moreSQL statements (e.g., one or more SQL queries) designed to access thedesired information. System data storage in system database 542 maygenerate query plans to access the requested data from database instance580.

In one embodiment, MTS 540 includes one or more application servers 550.From one perspective, application server 550 can be considered a networkinterface of MTS 540 to connect to network 530. Application server 550exchanges (i.e., receives and/or transmits) data with network 530, suchas receiving requests and sending replies or sending data. Applicationservers 550 may share hardware resources for interfacing with network530, or they may be assigned separate resources. In one embodiment, oneor more of application servers 550 can be implemented as an HTTPapplication server.

In one embodiment, each application server 550 is configured to handlerequests for any user associated with any organization that is a tenant.Thus, a request from user device 510 could be received and processed atany application server 550. There may be advantages to avoiding affinityfor a user and/or an organization or tenant to a specific applicationserver 550, such as the ability to add and remove application serversfrom a server pool at any time for any reason, as well as for workloadbalancing among the servers. In an implementation where user and/ortenant affinity is used, an application server could not be removedwithout completing its jobs and/or handing off users to another server.

In one embodiment, an interface system implementing a load balancingfunction (e.g., an F5 Big-IP load balancer) is communicably coupledbetween application servers 550 and the user devices to distributerequests to the application servers 550. In one embodiment, the loadbalancer uses a least connections algorithm to route user requests tothe application servers 550. Other examples of load balancingalgorithms, such as round robin and observed response time, also can beused. For example, in certain embodiments, three consecutive requestsfrom the same user could hit three different application servers 550,and three requests from different users could hit the same applicationserver 550. In this manner, MTS 540 is multitenant, wherein MTS 540handles storage of, and access to, different objects, data, andapplications across disparate users and organizations. In oneembodiment,

Each application server 550 includes elements to provide database accessservice and request processing. Application server 550 includes API(application programming interface) 552 and UI 554. UI 554 representsserver-side components that provide user interface elements that areprovided to user device 510 for display. API 552 provides an interfacefor users and/or developers to access resident processes of MTS 540.

In one embodiment, application server 550 includes application (appl)platform 560, which provides a sub-environment on which applicationshosted by application server 550 can be executed. Application platform560 may include an operating system or other control logic, as well asbusiness logic and common routines for use by the applications. Asillustrated, application platform 560 includes application setupmechanism 562 that supports creation and management of applications,including configuration, by application developers, which may be savedas metadata into tenant data storage of database (db) instance 580. Saveroutines 564 represent the mechanisms used to store data in databaseinstance 580, such as storing the application setup metadata. Suchapplications can be executed by subscriber users, for example, inprocess space 570.

In one embodiment, invocations to or related to such applications may becoded using PL/SOQL (Procedural Language Salesforce Object QueryLanguage) that provides a programming language style interface extensionto API 552. Thus, PL/SOQL 566 is capable of serving as a proceduralextension to an on-demand database centric service API that allows flowcontrol and transaction control to execute on a server in conjunctionwith database APIs (e.g., SOQL, data manipulation language (DML), orothers). PL/SOQL 566 can enable the capability to thread togethermultiple SOQL/DML statements as a single unit of work on the server.PL/SOQL 566 need not necessarily be considered a general purposeprogramming language, seeing that it may be implemented as heavily datafocused, but is not necessarily implemented that way. In one embodiment,PL/SOQL 566 can be used by developers to interlace with an on-demanddatabase system, in contrast to traditional application developers'conventional tools, such as PL/SQL (Structured Query Language) ofORACLE, Inc. of Redwood Shores, Calif., and others.

In one embodiment, PL/SOQL 566 includes variable and expression syntax,block and conditional syntax, loop syntax, object and array notation,pass by reference, and other syntax known to other programminglanguages. Thus, full control over syntax and the ability to referencedynamic schema elements is provided with a new language and runtime fordatabase services. Where embedded concepts that interface with on-demanddatabase applications are provided, syntax and semantics that are easyto understand and which encourage efficient use of database APIs mayalso be employed. In one embodiment, PL/SOQL 566 is implemented as astrong typed language with direct (non-quoted) references to schemaobjects such as Object and Field names (both standard and custom).

More details about PL/SOQL language embodiments is discussed in commonlyowned U.S. Provisional Patent Application 60/828,192 entitled,PROGRAMMING LANGUAGE METHOD AND SYSTEM FOR EXTENDING APIS TO EXECUTE INCONJUNCTION WITH DATABASE APIS, by Craig Weissman, filed Oct. 4, 2006,now expired, which is incorporated in its entirety.

In one embodiment, invocations to applications may be detected by one ormore system processes, which manage retrieving application metadata 586for the subscriber making the invocation and executing the metadata asan application in a virtual machine. Metadata 586 provides data relatedto access and/or use of data stored in database instance 580. In oneembodiment, metadata is stored in a separate table within databaseinstance 580, and in an alternative embodiment, metadata 586 is storedwith other data elements of user storage (such as with user storage 584of table 582-0.

In one embodiment, application server 550 includes process space 570,which may include tenant process spaces 576-0 through 576-N (for someinteger number N of process spaces configured in application server550), tenant management process space 572 and system process space 574.It will be understood that process space 570 is an abstraction toillustrate the resources allocated for execution of processes (e.g.,programs or applications) within application server 550. The skilledreader recognizes that memory and processor and other hardware resourcesmay need to be allocated, as well as software resources to support theexecution of a process. The processes may be executed as separatethreads, or may share a thread. In one embodiment, the number N oftenant processes is equal to a number of subscriber tenants. In anotherembodiment, the number N of tenant processes may be higher than thenumber of subscriber tenants. Tenant management process 572 providesmanagement of the other processes, including determining when certainprocesses execute. System process 574 executes operations related tofunctions of MTS 540.

Each application server 550 may be configured to tenant data storage indatabase instance 580 and the tenant data stored therein, and to systemdata storage of system database 542 and the system data stored thereinto serve requests of user devices. As mentioned above, in oneembodiment, tenant data is separated logically, and stored in the samemultitenant database. In one embodiment, database instance 580 storesdata in tables 582-0 through 582-M, where M is some integer number oftables. In one embodiment, different tables store data of differenttypes. Application metadata 586 may be implemented as a separate table.Alternatively, one of the tables 582-0 through 582-M could be a tablethat stores varying types of objects, which are defined through metadatastored in the table.

In one embodiment, database instance is further implemented with userstorage space distinct (e.g., identifiable) from its associated tenant.Thus, for example, user data may include the tenant ID, as well as anidentifier specific to a user. Thus, storage 584 may represent either orboth of tenant storage or user storage. For example, a copy of a user'smost recently used (MRU) items might be stored to in user storage withindatabase instance 580. Similarly, a copy of MRU items for an entireorganization that is a tenant might be stored to a tenant storage areaof database instance 580. In one embodiment, the tenant data and thesystem data (as illustrated by system database 542) are stored inseparate databases.

Application servers 550 may be communicably coupled to database systems,e.g., having access to system database 542 and tenant database instance580, via a different network connection. For example, one applicationserver may be coupled via a network (e.g., the Internet), anotherapplication server might be coupled via a direct network link, andanother application server might be coupled by yet a different networkconnection. The application servers may connect to the database systemsvia TCP/IP or another transport protocol, at least partially dependingon the network interconnect used.

Regarding storage in database instance 580, one tenant might be acompany that employs a sales force where each salesperson uses MTS 540to manage their sales process. Thus, a user might maintain contact data,leads data, customer follow-up data, performance data, goals andprogress data, and other data, all applicable to that user's personalsales process (e.g., storage 584, which may be tenant storage). Thus,all of the data and the applications to access, view, modify, report,transmit, calculate, or perform other operations can be maintained andaccessed via a user device having nothing more than network access. Inan example of an MTS arrangement, the user can manage his or her salesefforts and cycles from any of many different user devices. For example,if a salesperson is visiting a customer and the customer has a lobbywith Internet access, the salesperson can obtain critical updates as tothat customer while waiting for the customer to arrive in the lobby.

While each user's data might be separate from other users' dataregardless of the employers of each user, some data might beorganization-wide data shared or accessible by a plurality of users orall of the users for a given organization that is a tenant. Thus, theremight be some data structures managed by MTS 540 that are allocated atthe tenant level while other data structures might be managed at theuser level. Because MTS 540 may support multiple tenants includingpossible competitors, MTS 540 should have security protocols that keepdata, applications, and application use separate. Additionally, becausemany tenants may opt for access to an MTS rather than maintain their ownsystem, redundancy, up-time, and backup are additional functions thatmay be implemented in MTS 540. In addition to user-specific data andtenant specific data, MTS 540 may also maintain system level data usableby multiple tenants or other data. Such system level data might includeindustry reports, news, postings, and the like that are sharable amongtenants.

In one embodiment, each database instance 580 can be viewed as acollection of objects, such as a set of logical tables, containing datafitted into predefined categories. A “table” is one representation of adata object, and may be used herein to simplify the conceptualdescription of objects and custom objects according to the presentinvention. It should be understood that “table” and “object type” may beused interchangeably herein. Each table generally contains one or moredata categories logically arranged as columns or fields in a viewableschema. Each row or record of a table contains an instance of data foreach category defined by the fields.

For example, a CRM database may include a table that describes acustomer with fields for basic contact information such as name,address, phone number, fax number, or other information. Another tablemight describe a purchase order, including fields for information suchas customer, product, sale price, date, or other fields. In oneembodiment, a multitenant database has standard entity tables for use byall tenants. For CRM database applications, such standard entities mightinclude tables for Account, Contact, Lead, and Opportunity data, eachcontaining pre-defined fields. Thus, tables 582-0 through 582-M mayinclude standard, defined tables.

In one embodiment, tenants may be allowed to create and store customobjects, or they may be allowed to customize standard entities orobjects, for example by creating custom fields for standard objects,including custom index fields. U.S. patent application Ser. No.10/817,161, filed Apr. 2, 2004, entitled “Custom Entities and Fields ina Multi-Tenant Database System”, teaches systems and methods forcreating custom objects as well as customizing standard objects in amultitenant database system. In one embodiment, for example, all customentity data rows are stored in a single multitenant physical table,which may contain multiple logical tables per organization. It istransparent to customers that their multiple “tables” are in fact storedin one large table or that their data may be stored in the same table asthe data of other customers.

FIG. 6 represents a flow diagram of an embodiment of remote debugging ina server group environment. Flow diagrams as illustrated herein provideexamples of sequences of various process actions. Although shown in aparticular sequence or order, unless otherwise specified, the order ofthe actions can be modified. Thus, the illustrated implementationsshould be understood only as an example, and the process can beperformed in a different order, and some actions may be performed inparallel. Additionally, one or more actions can be omitted in variousembodiments of the invention; thus, not all actions are required inevery implementation. Other process flows are possible.

Client code is executed on a server group remote from the client thatowns the client code, 602. The client requests debug of the client code,604. A server from the server group processes the request, 606. In oneembodiment, the same server that identifies the request as a debugrequest will then perform debug session initiation operations. Inanother embodiment, a request identified as being for debug is passed toanother server of the group that is designated to handle such requests.

Thus, whether the server that processed the request and identified it asa debug request, or another server, establishes a communicationconnection to the client, initiates a debug session, and registers thedebug session, 608. The server that establishes the communicationconnection maintains the connection, or is part of a selected ordesignated subgroup of server that maintains the connection. It will beunderstood that flexibility in having a subgroup of servers to handlesuch communication connections also results in added overhead to manageand allocate responsibility among the subgroup.

At some point in the future after the debug session is established, theserver group receives a subsequent request related to the debug session,610. In one embodiment, the request is processed on whichever server isrunning and receives the request, 612. The processing server willconnect back to a server that maintains the communication connectionwith the client, which may be referred to as a “hold open” server.

When the hold-open server establishes the communication connection, inone embodiment, it requests breakpoints from the IDE client, which canbe sent back to the processing server. Thus, the processing serverobtains debug breakpoints from the client, 614, via the hold-openserver. The processing server executes the debug, which checks againstthe list of breakpoints upon executing the debug code, 616, for example,by checking the breakpoints against every Apex statement to run.

If a breakpoint is hit, 618, the processing server makes a callback tothe client via the hold-open server, 620. Thus, communication back tothe client related to the debug execution can be via a two-hopconnection (e.g., from the processing server to the hold-open server tothe client). In one embodiment, the first connection from the hold-openserver is a normal socket connection, and the second connection from theprocessing server is a proxied socket over a Comet HTTP connection. Withsuch a two-hop connection, any particular server can process any requestwithout overloading a particular server.

If a breakpoint was not reached, 618, or after the callback is made,620, the processing server continues to execute the debug, for as longas it is not complete. If the debug is not complete, 622, the executionand checking for breakpoints continues, 616. If the debug is complete,622, the execution is concluded, 624.

In alternate embodiments, one or more of the above details may bechanged regarding how different servers receive and process requests,and how communication is made back to the client. More particularly, inone embodiment, the receiving server of the group determines if itshould execute the request or whether the request should be sent toanother server for execution. In one embodiment, the receiving serversimply executes the request, and the receiving/processing serverdetermines what server maintains a connection back to the client.However, if the receiving server determines that it should not executethe request, it sends the request to another server for execution.

If it determines that it should execute the request or if the system isconfigured for the receiving server to execute requests received, itexecutes the request. Whichever server executes the request, itdetermines if it or another server of the server group maintains thecommunication connection to the client. If the executing servermaintains the connection and not another server, it communicates withthe client over the communication connection. If another server of thegroup maintains the connection, the processing server sends thecommunication to the other server that maintains the connection, whichthen communicates with the client.

Thus, in alternate embodiments, one or more server functions may becentralized with different servers (e.g., one server processes debugrequests, one server executes debug routines, or some other function).The potential downside to such an implementation is that one server mayend up with a work load that is not distributed among a group.

Various operations or functions are described herein, which may bedescribed or defined as software code, instructions, configuration,and/or data. The content may be directly executable (“object” or“executable” form), source code, or difference code (“delta” or “patch”code). The software content of the embodiments described herein may beprovided via an article of manufacture with the content stored thereon,or via a method of operating a communications interface to send data viathe communications interface. A machine readable medium or computerreadable medium may cause a machine to perform the functions oroperations described, and includes any mechanism that provides (i.e.,stores and/or transmits) information in a form accessible by a machine(e.g., computing device, electronic system, or other device), such asvia recordable/non-recordable storage media (e.g., read only memory(ROM), random access memory (RAM), magnetic disk storage media, opticalstorage media, flash memory devices, or other storage media) or viatransmission media (e.g., optical, digital, electrical, acoustic signalsor other propagated signal). A communication interface includes anymechanism that interfaces to any of a hardwired, wireless, optical, orother medium to communicate to another device, such as a memory businterface, a processor bus interface, an Internet connection, a diskcontroller. The communication interface can be configured by providingconfiguration parameters and/or sending signals to prepare thecommunication interface to provide a data signal describing the softwarecontent.

Various components described herein may be a means for performing theoperations or functions described. Each component described hereinincludes software, hardware, or a combination of these. The componentscan be implemented as software modules, hardware modules,special-purpose hardware (e.g., application specific hardware,application specific integrated circuits (ASICs), digital signalprocessors (DSPs), etc.), embedded controllers, hardwired circuitry,etc.

Besides what is described herein, various modifications may be made tothe disclosed embodiments and implementations of the invention withoutdeparting from their scope. Therefore, the illustrations and examplesherein should be construed in an illustrative, and not a restrictivesense. The scope of the invention should be measured solely by referenceto the claims that follow.

1-19. (canceled)
 20. A system comprising at least one memory devicecoupled with one or hardware processors, the system configurable to:receive a debugging session request utilizing an application programinterface (API) from a client device over a network connection, thedebugging session request to debug code executed by the server group;establish a connection between the server group and the client device toinitiate a debugging session between the server group and the clientdevice utilizing the API in response to the debugging session request,the server group to maintain open the connection with the client devicefor a duration of the debugging session; enable use of breakpoints viaremote debugging utilizing the client device; maintain state for thedebugging session with a resource shared among the server group; receiveservice requests via the API; and distribute the service requests amongthe server group for processing, wherein each server that processes arequest is to communicate via the API to the client device and whereinbreakpoint operations are modified to be registered in a same state asthe connection, and be connected back to the API.
 21. The system ofclaim 20, wherein the server group is to maintain state for thedebugging session by storing state for the debugging session in a sharedmemory cache shared among the servers of the server group.
 22. Thesystem of claim 20, wherein the server group is to further set abreakpoint that interrupts only execution of a thread of a serverhandling execution of the debugging session, and does not interruptother execution threads in the server group; wherein the breakpoint isset with an associated timeout, and execution of the thread is resumedand ended if activity on the thread ceases for the length of thetimeout.
 23. The system of claim 20, wherein the server group is tomaintain state for the debugging session by creating a thread associatedwith the debugging session and a stack associated with the thread,wherein the servers of the server group have access to the thread andthe stack.
 24. The system of claim 20, wherein a separate debuggingstack is created with an associated debugging thread started when thedebugging session is initiated.
 25. The system of thread 24, wherein thedebugging thread and associated stack are associated with a debuggingsession identifier (ID) and state information is to be persisted on thestack.
 26. The system of claim 20, wherein the client device furthercomprises an integrated development environment (IDE) to communicatewith the server group through the API.
 27. The system of claim 26,wherein the IDE enables use of the breakpoints on the client device. 28.The system of claim 20, wherein the server group further provides amultitenant database environment.
 29. The system of claim 20, whereineach server in the server group that processes a request is tocommunicate via a connection from a first server to the client device.30. A non-transitory computer-readable medium having stored thereoninstructions that, when executed by one or more processors within aserver group, are configurable to cause the one or more processors to:receive a debugging session request utilizing an application programinterface (API) from a client device over a network connection, thedebugging session request to debug code executed by the server group;establish a connection between the server group and the client device toinitiate a debugging session between the server group and the clientdevice utilizing the API in response to the debugging session request,the server group to maintain open the connection with the client devicefor a duration of the debugging session; enable use of breakpoints viaremote debugging utilizing the client device; maintain state for thedebugging session with a resource shared among the server group; receiveservice requests via the API; and distribute the service requests amongthe server group for processing, wherein each server that processes arequest is to communicate via the API to the client device and whereinbreakpoint operations are modified to be registered in a same state asthe connection, and be connected back to the API.
 31. The system ofclaim 30, wherein the server group is to maintain state for thedebugging session by storing state for the debugging session in a sharedmemory cache shared among the servers of the server group.
 32. Thesystem of claim 30, wherein the server group is to further set abreakpoint that interrupts only execution of a thread of a serverhandling execution of the debugging session, and does not interruptother execution threads in the server group; wherein the breakpoint isset with an associated timeout, and execution of the thread is resumedand ended if activity on the thread ceases for the length of thetimeout.
 33. The system of claim 30, wherein the server group is tomaintain state for the debugging session by creating a thread associatedwith the debugging session and a stack associated with the thread,wherein the servers of the server group have access to the thread andthe stack.
 34. The system of claim 30, wherein a separate debuggingstack is created with an associated debugging thread started when thedebugging session is initiated.
 35. The system of thread 34, wherein thedebugging thread and associated stack are associated with a debuggingsession identifier (ID) and state information is to be persisted on thestack.
 36. The system of claim 30, wherein the client device furthercomprises an integrated development environment (IDE) to communicatewith the server group through the API.
 37. The system of claim 36,wherein the IDE enables use of the breakpoints on the client device. 38.The system of claim 30, wherein the server group further provides amultitenant database environment.
 39. The system of claim 30, whereineach server in the server group that processes a request is tocommunicate via a connection from a first server to the client device.40. A method comprising: receiving a debugging session request utilizingan application program interface (API) from a client device over anetwork connection, the debugging session request to debug code executedby the server group; establishing a connection between the server groupand the client device to initiate a debugging session between the servergroup and the client device utilizing the API in response to thedebugging session request, the server group to maintain open theconnection with the client device for a duration of the debuggingsession; enabling use of breakpoints via remote debugging utilizing theclient device; maintaining state for the debugging session with aresource shared among the server group; receiving service requests viathe API; and distributing the service requests among the server groupfor processing, wherein each server that processes a request is tocommunicate via the API to the client device and wherein breakpointoperations are modified to be registered in a same state as theconnection, and be connected back to the API.
 41. The method of claim40, wherein the server group is to maintain state for the debuggingsession by storing state for the debugging session in a shared memorycache shared among the servers of the server group.
 42. The method ofclaim 40, further comprising: setting a breakpoint that interrupts onlyexecution of a thread of a server handling execution of the debuggingsession, and does not interrupt other execution threads in the servergroup; wherein the breakpoint is set with an associated timeout, andexecution of the thread is resumed and ended if activity on the threadceases for the length of the timeout.
 43. The method of claim 40,wherein the server group is to maintain state for the debugging sessionby creating a thread associated with the debugging session and a stackassociated with the thread, wherein the servers of the server group haveaccess to the thread and the stack.
 44. The method of claim 40, whereina separate debugging stack is created with an associated debuggingthread started when the debugging session is initiated.
 45. The methodof thread 44, wherein the debugging thread and associated stack areassociated with a debugging session identifier (ID) and stateinformation is to be persisted on the stack.
 46. The method of claim 40,wherein the client device further comprises an integrated developmentenvironment (IDE) to communicate with the server group through the API.47. The method of claim 46, wherein the IDE enables use of thebreakpoints on the client device.
 48. The method of claim 40, whereinthe server group further provides a multitenant database environment.49. The method of claim 40, wherein each server in the server group thatprocesses a request is to communicate via a connection from a firstserver to the client device.